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(57) Abstract 

A mouse mode manager for controlling 
operation of software objects on a display screen 
including a set of mouse mode buttons displayed 
on the display screen, the mouse mode buttons 
for selecting mutually exclusive operating modes 
of the software objects in response to a user 
supplied stimulus. To change the operating 
mode of the software objects a user stimulates 
the mouse mode button corresponding to the 
desired operating mode either by touching a 
touch sensitive screen over the desired mouse 
mode button or by pointing and clicking a mouse 
while the pointer is over the desired mouse mode 
button. The mouse mode manager stores the 
selected mouse operating mode in a strorage 
device and informs the software objects as to 
the current mouse mode status. 
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MOUSE MODE MANAGER AND VOICE ACTIVATION 
FOR NAVIGATING AND EXECUTING COMPUTER COMMANDS 

BACKGROUND OF THE INVENTION 
5 Technical Field of the Invention 

The present invention pertains in general to man 
machine interfaces and, in particular, to a method and 
apparatus for controlling and modifying the operation of 
software objects running on a computing device. 

10 P^sgription of Rel^t^d Art 

Personal computers and other computing devices often 
provide the user with a Graphical User Interface in 
conjunction with a mouse or other pointing device as a man 
machine interface. The use of a mouse allows the user to 

15 va2ry the operation of software objects displayed on the 
display screen. For example, moving a mouse over the 
object without clicking the mouse button typically results 
in a help message being displayed on the display screen. 
On the other hand, clicking the mouse button while over 

20 the software object results in activation of a 

configuration routine. Furthermore, moving a mouse over 
the software object, clicking the mouse button without 
releasing the mouse button, and then moving the mouse 
results in a drag-and-drop operation. 

25 On a touch sensitive display screen, however, there 

is no mouse with which to vary the operation of the 
software objects. Instead, the user touches the touch 
sensitive display screen over the software object and then 
removes their finger from the display screen to activate 

30 the software object in a manner akin to depressing the 
mouse button. Thus, the user is limited to defining the 
act of touching the touch sensitive display screen to a 
single operation. To overcome this limitation, users have 
either simply not been allowed to perform more than one 



aNSOOCID: <WO_99l2092At J_> 



wo 99/12092 



PCT/US98/18480 



-2- 

operation per software object or were required to use 
keystroke sequences which needed to be memorized. 

It would be advantageous therefore, to devise a 
method and apparatus for controlling and modifying the 
5 operating mode of software objects which are displayed on 

a display screen or running on a computing device which 
does not require the use of a mouse or keystroke 
sequences- It would further be advantageous if such a 
method and apparatus were suitable for use with either a 
10 touch sensitive screen, a mouse or voice activation. 

SUMMARY OF THE INVENTION 

The present invention comprises a mouse mode manager 
for controlling and modifying operation of software 

15 objects on a display screen or running on a computing 
device. The invention includes a set of mouse mode 
buttons displayed on the display screen, the mouse mode 
buttons for selecting mutually exclusive operating modes 
of the software objects and computer f\inctions in response 

20 to a user supplied stimulus. To chainge the operating mode 

of the software objects a user stimulates the mouse mode 
button corresponding to the desired operating mode either 
by touching a touch sensitive screen over the desired 
mouse mode button or by pointing and clicking the mouse 

25 while the pointer is over the desired mouse mode button 
or by voice activation. The mouse mode manager stores the 
selected mouse operating mode in a storage device and 
infoirms the software objects as to the current mouse mode 
status. In a second embodiment, the software objects are 

30 activated through voice commands. 

BRIEF DESCRIPTION OF THE DRAWINGS 

For a more complete understanding of the present 
invention, reference is made to the following detailed 
35 description taken in conjunction with the accompanying 

drawings wherein: 
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FIGURE 1 is a f\inctional block diagram of a mouse 
mode manager and a mouse mode panel executing on a 
c ompu t e r ; and 

FIGURE 2 is a flow diagram of a method for 
5 implementing a mouse mode manager; and 

FIGURE 3 is a functional block diagram of a voice 
activated system for executing commands on a computer. 

DETAILED DESCRIPTION OF THE INVENTION 

10 Referring now to Fig, 1, there is illustrated a 

functional block diagram of a mouse mode manager and a 
mouse mode panel executing on a computer. A computer 100 
comprises a processing unit 110, a storage device 120 and 
a display screen 13 0 including a touch sensitive display 

15 screen. A plurality of software objects 140 are stored 

in the storage device 120, executed on the processing xmit 
110 and displayed on the display screen 130. A plurality 
of software objects 141, such as a drop-and-drag manager, 
are also stored in the storage device 120 and are executed 

20 on the processing unit 110 but are not displayed on the 
screen 130. Furthermore, an operating system 142 is 
stored in the storage device 120 and is executed on the 
processing unit 110 . A mouse mode panel 150 is displayed 
on the display screen 13 0 and comprises a plurality of 

25 mouse mode buttons 160. In a preferred embodiment, the 
mouse mode panel 15 0 includes a normal operation button 
160A, a drag-and-drop mode button 160B, a help mode button 
160C, a configuration mode button 160D and an application 
specific button 160E. Mouse mode manager software is 

30 stored in the storage device 12 0 and executed on the 
processing unit 110. The mouse mode manager software 
creates the mouse mode panel 150 displayed on the display 
screen 130. 

Although the description of the preferred embodiment 
35 describes the use of a touch sensitive display screen 130, 
wherein, the user stimulates a mouse mode button 160 or 
a software object 140 or 141 or operating system 142 by 
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touching the display screen 13 0 with a finger or other 
object, it is understood that the present invention is 
also applicable when using other input devices such as a 
mouse or voice activation. Furthermore, it is understood 
5 that while the present invention controls and modifies 

operation of software objects displayed 140 and 141 
running on a computing device, the control and operation 
is performed in conjunction with the operating system 142 
in a manner consistent with the normal operation of a 

10 computing device. 

The mouse mode manager implements the mouse mode 
panel 150 in such a manner that only one mouse mode button 
160 is active at any given time and activation of one 
mouse mode button 160 results in the deactivation or 

15 locking out of all other mouse mode buttons 160. To 

activate the mouse mode buttons 160 the user touches the 
display screen 130 over the mouse mode button 160 
representative of the manner of operation desired. 

If the normal operation mode button 160A is active, 

20 stimulating the software object 140 by the user touching 
the display screen 130 over the software object 14 0 with 
no immediate release or by pointing and clicking the mouse 
over the software object 140 with no immediate release 
results in a bubble help message being displayed. 

25 Touching the display screen 130 over the software object 
14 0 followed by an immediate release or by pointing and 
clicking the mouse over the software object 140 followed 
by an immediate release or by speaking an associated 
command results in activation of the software object 140. 

30 If the drag-and-drop mode button 160B is active, 

stimulating the software object 14 0 by the user touching 
the display screen 130 over the software object 140 with 
no immediate release or by pointing and clicking the mouse 
over the software object 140 with no immediate release, 

35 results in the drag-and-drop manager being informed that 

the software object 140 is currently being dragged. While 
in this mode, the software object 140 moves in response 
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to movements of the finger touching the display screen 130 
or to movements in the mouse pointer. The software object 
140 drops when the finger releases from the display screen 
13 0 or when the mouse button is released. 
5 Touching the display screen 13 0 over the software 

object 140 followed by an immediate release or by pointing 
and clicking the mouse over the software object 140 
followed by an immediate release results in the drag-and- 
drop manager being informed that the software object 140 

10 is currently being dragged and the next software object 

capable of receiving the dragged software object 14 0 to 
be stimulated receives the software object 140. An 
example of this is when a software object is moved from 
one directory display to another directory display. In 

15 certain operations involving the drag-and-drop, the 

operating system 142 is also invoked. This is the case 
in the given example where the operating system 142 
updates the current memoary directory in which the software 
object 140 is listed, 

20 If the help mode button 160C is active, stimulating 

the software object 14 0 by the user touching the display 
screen 13 0 over the software object 140 followed by an 
immediate release or by pointing and clicking the mouse 
over the software object 140 followed by an immediate 

25 release or by speaking an associated command results in 

a full help message to be displayed on the display screen 
13 0. While the help message described activated in the 
normal operation mode displays a short message such as the 
name or operation of the software object, the help message 

30 displayed in the help mode is a more detailed message. 

If the configuration mode button 16 OD is active, 
stimulating the software object 140 by the user touching 
the display screen 130 over the software object 140 
followed by an immediate release or by pointing and 
35 clicking the mouse over the software object 140 followed 

by am. immediate release or by speaking an associated 
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command results in configuration options to be displayed 
on the display screen 130. 

Referring additionally now to Fig. 2, there is 
illustrated a flow diagram of a method for implementing 
5 a mouse mode manager. Each software object is either a 
subscription software object or a query software object. 
Subscription software objects are informed by the mouse 
manager whenever the mouse mode changes while query 
software objects make a request to the mouse mode manager 

10 to obtain the current mouse mode whenever the particular 
query software object has been stimulated. 

The mouse mode manager repeatedly monitors the mouse 
mode panel to determine if one of the mouse mode buttons 
has been newly activated (step 200) . If the mouse mode 

15 manager detects that one of the mouse mode buttons has 
been activated, the mouse mode manager stores the new 
mouse mode (step 210) and informs subscriber software 
objects of the new mouse mode (step 220) . If, on the 
other hand, the mouse mode manager fails to detect the 

20 activation of a mouse mode button in step 200 or after 
informing the subscriber software objects of the new mouse 
mode in step 220, the mouse mode manager determines 
whether one of the query software objects has requested 
to be informed of the mouse mode (step 230) . If there is 

25 a request, the mouse mode manager informs the requesting 
query software object of the mouse mode (step 240) and, 
in either event, the mouse mode manager returns to 
monitoring the mouse mode panel in step 200. 

A subscription software object repeatedly monitors 

30 to determine if the mouse mode manager has communicated 
a new mouse mode (step 245) . If a new mouse mode was 
received, the subscription software object stores the new 
mouse mode (step 247) . After storing the mouse mode or 
if no new mouse mode was received, the subscription 

35 software object determines if the software object has been 
stimulated (step 250) . If the software object has not 
been stimulated the software object returns to step 245 
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to determine if a new mouse mode is to be received^ 
otherwise, if the software object has been stimulated, the 
software object reads the stored mouse mode (step 260) . 
Based on the mouse mode, the software object acts 
5 accordingly (step 270) in response to object stimulation 
and then returns to motioning for a newly received mouse 
mode in step 245. 

A query software object repeatedly monitors to 
determine if the software object has been stimulated (step 

10 280) . If the software object has been stimulated, the 

software object requests the mouse mode from the mouse 
mode manager (step 290) . The query software object then 
receives the mouse mode from the mouse mode manager (step 
300) and based on the mouse mode, the software object acts 

15 accordingly (step 310) in response to object stimulation 

and then returns to motioning for software object 
stimulation in step 280. 

Referring additionally now to Fig. 3, there is 
illustrated a functional block diagram of a voice 

20 activated system for executing commands on a computer. 

A computer 100 comprises a processing unit 110, a storage 
device 120, a microphone 310 and a display screen 130. 
A plurality of software objects 14 0 are stored in the 
storage device 120, executed on the processing unit 110 

25 and displayed on the display screen 130. A plurality of 
software objects 141, such as a drop-and-drag manager, are 
also stored in the storage device 120 and are executed on 
the processing unit 110 but are not displayed on the 
screen 130. Furthermore, an operating system 142 is 

30 stored in the storage device 120 and is executed on the 
processing unit 110. A mouse mode panel 150 is displayed 
on the display screen 13 0 and comprises a plurality of 
mouse mode buttons 160. In a preferred embodiment, the 
mouse mode panel 150 includes a normal operation button 

35 160A, a drag-and-drop mode button 160B, a help mode button 

160C, a configuration mode button 160D and an application 
specific button 160E. Mouse mode manager software is 
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stored in the storage device 12 0 and executed on the 
processing unit 110. The mouse mode manager software 
creates the mouse mode panel 150 displayed on the display- 
screen 13 0. A list of voice commands and associated 
5 actions 300 to be performed on the software objects 140 

and 141 or the operating system 142 or the mouse mode 
panel 150 is stored in the storage device 120 . The 
processing unit monitors the microphone 310 to detect user 
spoken commands. When a spoken command is detected it is 

10 compared against voice commands stored in the list 300 
using any voice recognition system which are common in the 
industry. Upon detecting a match between the spoken 
command input through the microphone 310 and voice 
commands stored in the list 3 00, the processing unit 

15 executes the associated action. 

Although the preferred embodiment of the method and 
apparatus of the present invention has been described for 
use on a touch sensitive display screen, the method and 
apparatus is equally applicable to other embodiments using 

20 other pointing devices such as a mouse or light pen among 

others. Furthermore, while a preferred embodiment of the 
method and apparatus of the present invention has been 
illustrated in the accompanying Drawings and described in 
the foregoing Detailed Description, it is xinderstood that 

25 the invention is not limited to the embodiment disclosed, 

but is capable of numerous rearrangements, modifications, 
and sxibstitutions without departing from the spirit of the 
invention as set forth and defined by the following 
claims . 

30 
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WHAT IS CLAIMED IS: 

1. A method for selecting a mouse operation mode 
of software objects on a display screen comprising: 

displaying mouse mode buttons on a display 
5 screen, the mouse mode buttons representative of mouse 
operation modes of the software objects, the mouse mode 
buttons for selecting mutually exclusive operation modes 
of the software objects in response to a stimulus; 

detecting stimulation of the mouse mode buttons; 
10 storing the mouse operation mode; and 

informing the software objects as to the mouse 
operation mode. 

2, The method recited in claim 1, wherein the step 
15 of informing the software objects as to the mouse 

operation mode comprises communicating the mouse operation 
mode to subscriber software objects in response to changes 
in the mouse operation mode. 

20 3 . The method recited in claim 1 , wherein the step 

of informing the software objects as to the mouse 
operation mode further includes the steps of : 

detecting a mouse operation mode recjuest from 
a query software object; and 

25 commiinicating the mouse operation mode to the 

query software object in response to the detected request, 

4. The method recited in claim 1, wherein the step 
of detecting stimulation of a mouse mode button further 

30 includes detecting stimulation of a normal operation mouse 
mode button. 

5. The method recited in claim 1, wherein the step 
of detecting stimulation of a mouse mode button further 

35 includes detecting stimulation of a drop-and-drag 
operation mouse mode button . 
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6. The method recited in claim 1, wherein the step 
of detecting stimulation of a mouse mode button further 
includes detecting stimulation of a help operation mouse 
mode button. 

5 

7. The method recited in claim 1, wherein the step 
of detecting stimulation of a mouse mode button further 
includes detecting stimulation of a configuration 
operation mouse mode button. 

10 

8. The method recited in claim 1, wherein the step 
of detecting stimulation of a mouse mode button further 
includes detecting stimulation of an application specific 
operation mouse mode button. 

15 

9 . A method for changing operation of a 
subscription software object displayed on a display- 
screen comprising the steps of : 

receiving a mouse operation mode from a mouse 
20 mode manager in response to stimulation of a mouse mode 
button; 

storing the mouse operation mode in the 
subscription software object; 

detecting stimulation of the subscription 
25 software object; 

operating the subscription software object in 
a manner corresponding to the mouse operation mode. 

10. The method recited in claim S, wherein the step 
30 of receiving a mouse operation mode further includes 

receiving 

a normal operation mouse mode. 

11 . The method recited in claim 9 , wherein the step 
35 of receiving a mouse operation mode further includes 

receiving 

a help operation mouse mode. 
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12. The method recited in claim 9, wherein the step 
of receiving a mouse operation mode further includes 
receiving 

a drag-and-drop operation mouse mode. 

5 

13. The method recited in claim 9, wherein the step 
of receiving a mouse operation mode further includes 
receiving 

a configuration operation mouse mode. 

10 

14. The method recited in claim 9, wherein the step 
of receiving a mouse operation mode further includes 
receiving 

an application specific operation mouse mode. 

15 

15 . A method for changing operation of a query 
software object displayed on a display screen comprising 
the steps of : 

detecting stimulation of the qpaery software 

20 object; 

requesting a mouse operation mode from a mouse 
mode manager ; 

receiving the mouse operation mode; 
operating the query software object in a manner 
25 corresponding to the mouse operation mode. 

16. The method recited in claim 15, wherein the step 
of receiving a mouse operation mode further includes 
receiving a normal operation mouse mode, 

30 

17. The method recited in claim 15, wherein the step 
of receiving a mouse operation mode further includes 
receiving a help operation mouse mode . 

35 18. The method recited in claim 15, wherein the step 

of receiving a mouse operation mode further includes 
receiving a drag-and-drop operation mouse mode. 
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19. The method recited in claim 15, wherein the step 
of receiving a mouse operation mode further includes 
receiving a configuration operation mouse mode. 



5 20. The method recited in claim 15, wherein the step 

of receiving a mouse operation mode further includes 
receiving an application specific operation mouse mode. 

21. A mouse mode manager for selecting a mouse 
10 operation mode of software objects on a display screen 

comprising: 

a set of mouse mode buttons displayed on the 
display screen, the buttons for selecting mutually 
exclusive operation modes of the software objects in 
15 response to a stimulus; 

a storage device for storing a current mouse 
mode status; and 

means for informing the software objects as to 
the current mouse mode status . 

20 

22. Apparatus for activating software objects using 
voice commands comprising: 

a storage device for storing a list of voice 
commands and associated actions; 
25 a microphone for inputting spoken commands; and 

a processing unit for detecting spoken commands 
input through the microphone and matching the spoken 
command with voice commands stored in the storage device, 
the processing unit further for executing the action 
30 associated with the voice command. 
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